Description:

As a Data Engineer in the Lakehouse and AI Data Platform team, you will design, build, test, and support data pipelines and curated datasets on the firm’s modern data platform. You will work across ingestion, transformation, modeling, optimization, and data quality, helping to deliver data products that are reliable, scalable, and fit for purpose. Where there are gaps in platform functionality, you may also contribute to shared tooling or framework components that improve how the platform is used and operated.

Key Technologies:

ANSI SQL, Apache Spark, Kafka, JSON, Avro, Parquet, Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ, Python, Java, CI/CD tooling, containerized, Kubernetes

Requirements:

• 3+ years of experience with a Bachelor’s or Master’s degree in a relevant discipline, or equivalent practical experience displaying strong quantitative or data engineering skills.
• Strong hands-on programming experience in Python or Java. Good working knowledge of SQL, including troubleshooting, optimization, and data analysis.
• Build, enhance, support, and refactor batch and streaming data pipelines on the Lakehouse and AI data platform. Ensure pipelines are production-ready and well-tested.
• Develop raw, refined, and curated datasets. Apply temporal data modeling (handling historical state/change over time), schema design, schema evolution, partitioning, and clustering. Make design choices across normalized vs. denormalized models and natural vs. surrogate keys.
• Implement controls to validate the completeness, accuracy, and consistency of data. Use reconciliation approaches to investigate production breaks.
• Familiarity with version control, testing, release discipline, and CI/CD practices.
• Demonstrate ownership of technical design across multiple datasets, guide implementation standards within the team, lead delivery for a workstream, and support less experienced engineers.